Using Feature Selection to Find Inputs That Work Better as Extra Outputs

In supervised learning there is usually a clear distinction between inputs and outputs | inputs are what you measure, outputs are what you predict from those measurements. The distinction between inputs and outputs is not this simple. Previously, we demonstrated that on synthetic problems some input features are more useful when used as extra outputs than when used as inputss6]. This paper shows the same eeect on a real problem, and presents a means of determining what features can be used as extra outputs. We show that the feature selection method devised by Koller and Sahamii11] can be used to select features to use as extra outputs, and that using some features as as extra outputs instead of as inputs yields better performance on the DNA splice-junction domain. 1 MOTIVATION The goal in supervised learning is to learn functions that map inputs to outputs with high predictive accuracy. The common practice in backprop nets is to use all features that will be available for test cases as inputs, and use as outputs only features that need to be predicted. On real problems, where there may be many redundant or irrelevant features, using all the available features as inputs is often suboptimal. Many algorithms learn better given a carefully selected subset of the features to use inputss3, 10, 11]. If feature selection is used to nd the features to use as inputs, what should be done with the features not selected? Usually, features not selected for use as inputs are discarded. But, there are other ways to beneet from features without using them as inputs. One way to beneet from features not used as inputs is multitask learning. Multitask learning (MTL) is an inductive transfer method where extra tasks are learned in parallel with the main task while using a shared representation. Because the extra tasks share a hidden layer with the main task, internal representations learned for the extra tasks can be used by the main task outputs, often improving performance on the main task. MTL in backprop nets is well documentedd13, 1, 2, 4, 8, 9, 7]. Most applications of MTL are to problems where some features available for the training set will not be available for future test casess5]. We recently demonstrated that there are problems where some features that could be used as inputs would be 1